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Executive Summary 

The 45th Weather Squadron (45 WS) Launch Weather Officers (LWOs) use the 12-km resolution 
North American Mesoscale (NAM) model (MesoNAM) forecasts extensively to support launch weather 
operations. However, the actual performance of the model at Kennedy Space Center (KSC) and Cape 
Canaveral Air Force Station (CCAFS) has not been measured objectively. In order to have tangible 
evidence of model performance, the 45 WS tasked the Applied Meteorology Unit (AMU) to conduct a 
detailed statistical analysis of model output compared to observed values. The analysis compared the 
MesoNAM forecast winds, temperature and dew point to the observed values from the sensors on 
specified KSC/CCAFS wind towers. 

The AMU calculated the following statistics to verify the performance of the model: 

• Bias (mean difference), 

• Standard deviation of Bias, 

• Root Mean Square Error (RMSE), and 

• Hypothesis test for Bias = 0. 

The AMU calculated the verification statistics once the files were properly formatted and stratified. 
First the difference between all hourly model forecasts and tower observations was calculated and then 
the mean monthly bias, the standard deviation of the mean monthly bias and then the RMSE. Finally, the 
hypothesis zero test was applied to the mean monthly bias statistics. 

The model’s performance statistics were calculated for each wind tower based on sensor height and 
model initialization time. The data were stratified by month and onshore/offshore wind direction based on 
the orientation of the coastline to each tower’s location prior to the statistics calculations. 

The 45 WS requested the AMU only calculate statistics for the current MesoNAM which became 
operational in mid-August 2006 and was based on the Weather Research and Forecasting model. 
Therefore the period of record for the data used in this task began with the first full month of data, 
September 2006, through May 20 1 0. 

The model performance indicated: 

• The accuracy decreased as the forecast valid time from the model initialization increased, 

• There was a diurnal signal in temperature with a cool bias during the late night and a warm bias 
during the afternoon, 

• There was a diurnal signal in dew point temperature with a low bias during the afternoon and a 
high bias during the late night, and 

• The model parameters at each vertical level most closely matched the observed parameters at 
heights closest to those vertical levels. 

The AMU developed a graphical user interface (GUI) that consists of a multi-level drop-down menu 
written in JavaScript embedded within Hyper-Text Markup Language code that can be used with most 
popular web browsers. This GUI allows the LWO to easily and efficiently navigate among the charts and 
spreadsheet files. 

The objective statistics provide the forecasters knowledge of the model’s strengths and weaknesses 
and the GUI allows quick access to the data which will result in improved forecasts for operations. 
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1. Introduction 


The 45th Weather Squadron (45 WS) Launch Weather Officers (LWOs) use the 12-km resolution 
North American Mesoscale (NAM) model (MesoNAM) text and graphical product forecasts extensively 
to support launch weather operations. However, the actual performance of the model at Kennedy Space 
Center (KSC) and Cape Canaveral Air Force Station (CCAFS) has not been measured objectively. In 
order to have tangible evidence of model performance, the 45 WS tasked the Applied Meteorology Unit 
(AMU; Bauman et al. 2004) to conduct a detailed statistical analysis of model output compared to 
observed values. 

The model products are provided to the 45 WS by ACTA, Inc. and include hourly forecasts from 0 to 
84 hours based on model initialization times of 00, 06, 12 and 18 UTC. The objective analysis compared 
the MesoNAM forecast winds, temperature (T) and dew point (T d ), as well as the changes in these 
parameters over time, to the observed values from the sensors in the KSC/CCAFS wind tower network 
shown in Table 1. The LWOs support launch and landing activities using the sensors in the column 
labeled “ Required Sensor Heights The other sensors present on the towers are shown in the column 
labeled " Additional Sensor Heights" and were included in this work for completeness. The objective 
statistics calculated will give the LWOs knowledge of the model’s strengths and weaknesses based on the 
observations at individual wind towers, which will result in improved forecasts for operations. 


Table 1. Towers, launch activities and sensor heights at KSC and CCAFS used in the objective 
analysis to verify the MesoNAM forecasts. 

Tower Number 

Supported Activity 
and Facility 

Required Sensor 
Heights 

Additional Sensor 

Heights 

002 

Delta II (LC-17) 

6 ft, 54 ft, 90 ft 

145 ft, 204 ft 

006 

Delta IV (LC-37)/Falcon 9 (LC-40) 

54 ft 

6 ft, 12 ft, 162 ft, 204 ft 

108 

Delta IV (LC-40)/Falcon 9 (LC-40) 

54 ft 

6 ft, 12 ft 

110 

Atlas V/Falcon 9 (LC-40) 

54 ft, 162 ft, 204 ft 

6 ft, 12 ft 

041 

Atlas V (LC-41) 

230 ft 


393/394 

Shuttle (LC-39A) 

60 ft 


397/398 

Shuttle (LC-39B) 

60 ft 


511/512/513 

Shuttle Landing Facility 

6 ft, 30 ft 
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2. Data 


The period of record (POR) for the data set was September 2006 - January 2010. The AMU chose 
September 2006 as the beginning of the POR because the 12-km Weather Research and Forecasting 
(WRF) model became the operational MesoNAM model in mid-August 2006 and the 45 WS required the 
AMU conduct the model verification using the “current operational model”. 

2.1 Tower Observations 

The AMU used the KSC/CCAFS wind tower data from the AMU archive and the AMU wind tower 
quality control (QC) software to remove erroneous observations from the dataset. Scripts were written in 
S-PLUS® (Insightful Corporation 2007) to import and modify the QC’d wind tower observation files to 
remove unneeded time periods, observed parameters and sensor heights from the dataset for each tower; 
and to reformat the files to prepare them for merger with the MesoNAM forecast files. The locations of 
the towers used for the verification are shown on the map of KSC/CCAFS in Figure 1 . 



Figure 1 . Map of KSC/CCAFS showing the locations of the wind towers used to verify MesoNAM 
forecasts (red pentagons labeled with tower number and the supported launch activity), the locations of 
the MesoNAM model grid points (green circles with black dot) and CCAFS weather station (magenta 
square labeled KXMR). The MesoNAM land/sea mask is shown by the magenta line. The model assumes 
only land is to the left of the land/sea mask. 

Since the tower data were reported every 5 minutes and the MesoNAM forecasts were hourly, the 
45 WS requested the AMU calculate the mean value for each observed parameter from the tower data at 
the top of every hour using the observations from 30 minutes prior and 30 minutes after the hour. The 
S-PLUS scripts were written to reformat the tower data and calculate the mean values in this manner. 
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2.2 MesoNAM Forecast Products 


The AMU requested and obtained the archived MesoNAM forecasts from ACTA, Inc. The 
MesoNAM forecast files were provided to the AMU as space-delimited text files. Each file contained the 
initialization and hourly forecasts to 84 hours at a single model initial time of 00, 06, 12 or 18 UTC. The 
data were valid at a single point extracted from the model forecasts, identified as "KXMR" in each file. 
This represents the location of the CCAFS weather station, which is located near the center of CCAFS 
and is identified in Figure 1 by the magenta square below the CCAFS label. The closest model grid point, 
which represents the point data used by the 45 WS, is located 3.1 NM southwest of KXMR over the 
Banana River. It is shown in Figure 1 as a green circle with a black dot and labeled "NAM". Because the 
grid point is to the left of the model land/sea mask, the model assumes the point is over land. 

An inventory of the MesoNAM files in the POR revealed 128 missing files, or model runs, out of a 
possible 4996 files for the 1249 days. Some days were missing less than four model runs while others 
were missing all four model runs. The result was a total of 1216 days containing at least one model run. 

2.3 Data Stratification 

The 45 WS requested the results be stratified by 45°, 90°, 180° and 360° wind sectors, year, warm 
season (May-September), cool season (October-April), month and model initialization time. They also 
requested the sector directions be oriented to discriminate between onshore/offshore flow for each tower 
and to analyze the model forecasts only for the current operational version of the MesoNAM. 

After discussions with the 45 WS, the requirement for yearly, warm season and cool season 
stratifications was rescinded since they believed the monthly stratification would be a better time period 
to support launch and landing operations. Based on this determination, starting with the smallest wind 
sector stratification of 45°, the AMU calculated bias statistics for three towers to determine if the sample 
size was sufficient to be statistically significant, and found it was not. The same was done for the 90° 
wind sector stratification and the sample size was also too small. Testing a wind sector stratification of 
1 80° indicated the results would produce sufficient sample sizes to calculate monthly statistics. After the 
AMU provided these test results to the 45 WS, they decided the best 180° stratification for each tower 
would be calculated as onshore and offshore based on the orientation of the coastline to each tower’s 
location. 
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3. File Formats 

A significant amount of file format manipulation was required in order to compute the required 
statistics between the observation and model forecast files. In both the observation and forecast data, there 
were more parameters in the files than required to compute the verification statistics. In order to more 
efficiently compute the statistics for this large data set, the AMU removed the unneeded parameters, 
which reduced file sizes, and then reformatted the data within the spreadsheets to reduce the complexity 
of the task. The goal was to combine the observed and forecast data into Excel format files to compute the 
statistics and provide the results in a familiar format to the LWOs. 

3.1 Tower Observation Files 

The raw tower observation files were available in space-delimited text file format. Each file 
represented one tower and included all sensor heights at all locations (NW and SE side of tower) on that 
tower. As shown in Figure 2, the format of the files included columns containing, in order left to right, 
date (year, month, day), time (hour, minute, second), tower identifier (3-digit number), sensor location, 
sensor height (ft), mean wind direction, mean wind speed (ms 1 ), peak wind direction, peak wind speed 
(ms 1 ), T (°C), T d (°C) and relative humidity (%). 


Fa 

A B C D 

L E 

F e H 1 J K J_ L 

u 

Year/Month/Day Time (HMS) Tower ID Sensor Loc 

Sensor Hgt 

Direction Spd (m/s) PDirection PSpd (m/s) T (C) Td (C) RH (%) 


Figure 2. Depiction of column headers for tower observation data contained in the raw data files. 

After running the AMU QC program, the tower files were reformatted and output as space-delimited 
text files and each file contained all sensor heights on a tower on one side of the tower (NW or SE). 
Figure 3 shows the format of the new QC files that included columns containing, in order left to right, 
tower identifier (3-digit number), year, day, hour, minute (5-minute intervals), sensor height (ft), T (°F), 
T d (°F), mean wind speed (kt), mean wind direction, peak wind speed (kt), peak wind direction and 
relative humidity (%). 


A 

A T B C 

D 

E F G L 

H 

1 J K L M 

N 

1 

Tower ID Year Month 

Day 

Hour Minute Sensor Hgt 

T(F) 

Td (F) Spd (kt) Direction PSpd(kt) PDirection 

RH (*) 


Figure 3. As in Figure 2 but for the tower observation data as formatted by the AMU QC program. 


The S-PLUS scripts mentioned in Section 2.1 were run on the QC’d files to calculate the mean values 
of wind speed, wind direction, T and T d at the top of every hour using the observations from 30 minutes 
prior and 30 minutes after the hour and to output the resulting files in Excel format - one file for every 
month for each year in the POR. Each file contained data for one sensor height at one sensor location per 
tower. Figure 4 shows the format of the files with columns containing, in order left to right, year, month, 
day, hour, mean wind speed (kt), mean wind direction, mean T (°F) and mean T d (°F). In preparation for 
merger with the MesoNAM forecast files, the AMU wrote Microsoft Excel Visual Basic for Applications 
(VBA) scripts to process the hourly mean files into Excel workbook files containing four spreadsheets per 
workbook. Each spreadsheet represented the MesoNAM model initialization times of 00, 06, 12 and 18 
UTC and contained 84 hourly mean observations to match the 84 hours of model forecasts per model 
initialization. 


A 

A 

B 

C 

D 

E 

F G H 

i| 

Year 

Month | 

Day 

Hour 

Mean Spd (kt) 

Mean Dir [ Mean T (F) | Mean Td (F) | 


Figure 4. Column format for observation data from an individual 
tower sensor after hourly mean values were calculated and the 
files were exported to Excel format. 
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3.2 MesoNAM Forecast Files 


The AMU wrote VBA scripts to import the MesoNAM files into Excel spreadsheets and reformatted 
them to merge with the wind tower observation spreadsheets. As shown in Figure 5, the format of the raw 
MesoNAM files included columns containing, in order left to right, date (year, month, day/hour, minute), 
T (°C), T d (°C), wind speed (kt), wind direction, mean sea level pressure (mb), low cloud (% coverage), 
mid cloud (% coverage) and high cloud (% coverage). 


A 

A 

B 

■ C 

U- 

UJ 

O 

G 

" H 

i ] 

1 

YYMMDD/HHMM 

T(C) 

Td (C) 

Spd (kt) Direction PMSL 

LCLD 

MCLD 

HCLD 


Figure 5. Depiction of column headers for MesoNAM forecast data 
contained in the raw data files. 


Reformatting the raw MesoNAM files included converting the temperature and dew point from 
Celsius to Fahrenheit and moving rows and columns in the MesoNAM spreadsheets to merge with the 
wind tower spreadsheets. Figure 6 shows the final format of the column headers depicting the model 
forecast data that was used in the statistical verification containing, in order left to right, date (year, 
month, day, hour), wind speed (kt), wind direction, T (°F), T d (°F) 


A 

A 

B 

C 

D 

E 

i'l 

YYMMDDHH 

WRF Spd 

WRF Dir 

WRF T 

WRF Td 


Figure 6. As in Figure 5 but for the 
MesoNAM forecast data after final 
reformatting. 


3.3 Merged Files 

The AMU wrote VBA scripts to create an Excel workbook for each of the 1216 days with at least one 
model run. The scripts opened the tower observation Excel files and MesoNAM Excel files and merged 
them to create workbooks that included up to four spreadsheets per workbook, one for each available 
model run, containing the combined tower observations and MesoNAM forecasts for each sensor on 
every tower. This resulted in a total of 57,124 workbooks. As shown in Figure 7, the format of the merged 
files included columns containing, in order left to right, year, month, day, hour, mean wind speed (kt), 
mean wind direction, mean T (°F) and mean T d (°F) of the tower observations and date (year, month, day, 
hour), wind speed (kt), wind direction, T (°F), T d (°F) of the MesoNAM forecasts. 


A 

A 

B 

C 

D 

E 

F 

G 

H 

1 J 

K 

L 

M 

N 

l| 

Year 

1 Month 

Day 

Flour 

] Mean Spd 

Mean Dir 

Mean T 

Mean Id | 

| YYMMDDHH 

WRF Spd 

WRF Dir 

WRF T 

WRF Td | 


Figure 7. Depiction of column headers for files containing merged tower observation data and 
MesoNAM forecast data. The observation data was located in columns A-H and the MesoNAM data was 
located in columns J-N. 
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4. Verification Statistics 

The AMU calculated the verification statistics once the files were properly formatted and stratified. 
First the difference between all hourly model forecasts and tower observations was calculated and then 
the mean monthly bias, the standard deviation of the mean monthly bias and then the root mean square 
error (RMSE). Finally, the hypothesis zero test was applied to the mean monthly bias statistics. 

4.1 Statistics 

The 45 WS requested the AMU calculate the following statistics to verify the performance of the 
model against the observations: 

• Bias (mean difference), 

• Standard deviation of Bias, 

• Root Mean Square Error (RMSE), and 

• Hypothesis test for Bias = 0. 

4.2 Difference Calculations 

The model “bias” was calculated for each model forecast against every observation, which 
represented the difference between the MesoNAM forecast parameters and the tower observation 
parameters. This difference was calculated by subtracting the observed parameter from the model forecast 
for each hour of every day in the POR stratified by tower, sensor height and onshore/offshore flow as 
shown by the equation below. 

DifMesoNAM = Model Forecast - Tower Observation 

4.2.1 NW and SE Sensors 

Before making the difference calculations, the AMU needed to stratify the data into onshore and 
offshore components. Special care was required when stratifying the observations at Towers 002, 006 and 
1 1 0 because they have dual sensors located on the N W and SE side of each tower at the same height on 
the tower. Figure 8 depicts which sensor is chosen to report real-time observations to the forecaster 
display based on the observed wind direction. The NW sensor is always used when the winds are between 
249° and 22° (thick light red arc) and the SE sensor is always used when the winds are between 69° and 
203° (thick light green arc). However, both sensors can display observations in the gray arcs, 23° to 68° 
or 204° to 248°. The sensor used in the gray arc is based on the last sensor used before the winds were in 
the gray arc. For example, if the SE sensor was the previously selected sensor based on wind direction, 
then that sensor will continue to be the selected sensor until winds are observed between 248° and 23°. 
Conversely, if the NW sensor was the previously selected sensor, it will remain so until the winds are 
observed between 68° and 204°. Data from both sensors are saved for archive regardless of wind 
direction. For this task, the NW sensor was used for winds between 226° and 45° (thin red arc in Figure 
8) and the SE sensor was used for winds between 46° and 225°(thin green arc in Figure 8). 
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NW Sensor 


SE Sensor 


270° 


248° 

225 


204° 


Figure 8. Dual-sensor configuration on Towers 002, 006 and 110. 
Sensors are mounted on the NW and SE side of each tower. The NW 
sensor is always selected when the winds are between 249° and 22° 
(light red arc) and the SE sensor is always selected when the winds are 
between 69° and 203° (light green arc). Either sensor can be used in the 
two gray regions when the winds are between 23° and 68° or 204° and 
248°. 


4.2.2 Onshore/Offshore Directions 

Figure 9 shows the locations of Towers 002, 006 and 110 (identified by the green ellipses). The 
coastline orientation in the vicinity of Tower 002 is northeast to southwest. Therefore, the onshore 
stratification only used data from the SE sensor and the offshore stratification only used data from the 
NW sensor. Towers 006 and 1 10 are affected by a coastline orientation from NW to SE. Therefore, their 
onshore stratification required using the NW sensor for winds from 316° to 45° and the SE sensor for 
winds from 46° to 135°. Conversely, their offshore stratification required using the NW sensor for winds 
from 226° to 315° and the SE sensor for winds from 136° to 225°. To summarize the sensor 
configurations for onshore and offshore flow for the dual-sensor towers: 

For Tower 002: 

• Onshore: SE sensor 46° to 225° 

• Offshore: NW sensor 226° to 45° 

For Towers 006 and 1 10: 

• Onshore: NW sensor 3 1 6° to 45° and SE sensor, 46° to 135° 

• Offshore: NW sensor 226° to 3 1 5° and SE sensor 1 36° to 225° 

The AMU developed VBA scripts that select the appropriate sensors for the onshore and offshore 
calculations for the dual-sensor towers. 
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Figure 9. Map of KSC/CCAFS showing the locations of the wind towers used to verify MesoNAM 
forecasts (red pentagons labeled with tower number and the supported launch activity) and the dual- 
sensor towers (green ellipses). The onshore and offshore stratifications were determined by the coastline 
alignment relative to the Atlantic Ocean for each tower. 

The wind tower configurations at the Shuttle pads presented a similar challenge to computing onshore 
and offshore statistics as for the dual-sensor towers. As Figure 9 shows. Tower 393 (Pad 39A) and Tower 
397 (Pad 39B) are on the NW side of each pad and have sensors mounted on the NW side of each tower. 
Tower 394 (Pad 39A) and Tower 398 (Pad 39B) are on the SE side of each pad and have sensors mounted 
on the SE side of each tower. This sensor configuration is similar to that described above for the dual- 
sensor towers except the sensors are not on the same tower. The AMU also developed VBA scripts to 
extract the appropriate data from each of these towers and then combine the data at each pad to compute 
statistics for the onshore and offshore components. 

4.2.3 Onshore/Offshore Differences 

Using the Excel files with the merged tower observations and MesoNAM forecasts, the AMU- 
developed VBA scripts calculated the difference of wind speed, wind direction, T and T<j based on the 
onshore and offshore wind direction stratifications. The scripts opened a merged Excel workbook file, 
calculated the difference and entered the results in columns P-S. Figure 10 shows an example of an 
offshore stratification worksheet from Tower 002. The difference was not calculated in rows 23-25 and 
47-51 because the mean wind direction was onshore at those times, as shown in the green ellipses in 
column F. 
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Figure 10. Example of a merged Excel workbook file showing the tower observations from Tower 002 
(columns A-H), MesoNAM forecasts (columns J-N) initialized at 00 UTC (00Z tab at bottom left), and 
difference calculations (columns P-S) for an offshore stratification 

The procedures to calculate the difference included quality checks and stratification calculations. In the 
VBA scripts, the first step implemented was a QC test that ensured the tower observations did not exceed 
unrealistic values and/or climatological maxima and minima based on a 30-year Operational Climatic 
Data Summary provided by the 14th Weather Squadron for the Shuttle Landing Facility (KTTS). Even 
though an initial QC was performed on the raw observational data, through visual inspection, the AMU 
discovered some out-of-bounds observations were still present after the initial QC. The VBA script QC 
test checked to ensure the observations were within the following ranges: 

• Mean wind direction > 0° and < 360°, 

• Mean wind speed > 0 kt and < 65 kt, 

• Mean T > 20 °F and < 1 00 °F, and 

• Mean T d > 5 °F and < 85 °F. 


4.2.4 Coastline Orientation 

To stratify the onshore and offshore flow, the AMU scripts accounted for the orientation of the 
coastline differences between Tower 002 and the rest of the towers. For T and T d sensors without 
corresponding wind sensors at the same height, the wind sensors closest to the T and T d sensors were used 
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to determine the stratification for those observations. For example, the 12 ft wind sensors were used for 
the 6 ft T and T d sensors. 

To calculate the proper stratification for Tower 002, the AMU scripts checked the mean wind direction 
associated with each parameter prior to every difference calculation. Separate calculations were made for 
the sensors on the NW and SE sides of the tower. For onshore flow, the scripts verified the mean wind 
direction observed was > 45° and < 225°, represented by the green arrows and arc in Figure 11. In this 
case, data from the SE sensor were used. For offshore flow, the NW sensor was used if the scripts verified 
the mean wind direction observed was > 225° and < 360° or < 45°, represented by the red arrows and arc 
in Figure 1 1. 



Figure 11. Representation of the sensor configuration for 
onshore and offshore stratifications on the dual-sensor Tower 
002. The NW sensor is used for offshore flow stratification while 
the SE sensor is used for onshore flow stratification. 

To calculate the proper stratification for the other two dual-sensor towers, Towers 006 and 110, the 
AMU scripts also checked the mean wind direction associated with each parameter prior to every 
difference calculation and made separate calculations for the sensors on the NW and SE sides of the 
tower. Because of the towers’ orientation to the coastline, sensors on either side could be used for both 
onshore and offshore stratifications, which required using NW and SE sensors to determine the 
stratification. For onshore flow, the NW sensor was used if the mean wind direction was > 3 15° or < 45°, 
represented by a green arrow and thick light green arc in Figure 12. The SE sensor was used if the mean 
wind direction was > 45° and < 135°, represented a green arrow and thick dark green arc in Figure 12. For 
offshore flow, the NW sensor was used if the mean wind direction was > 225° and <315°, represented by 
a red arrow and thick light red arc in Figure 12. The SE sensor was used if the mean wind direction was 
> 135° and < 225°, represented by the red arrow and thick dark red arc in Figure 12. 



Figure 12. As in Figure 1 1 but for the dual-sensor Towers 006 
and 1 10. Both the NW and SE sensors must be used for onshore 
and offshore flow stratifications. The gray dashed line represents 
the divide between the data used from the NW sensor and from 
SE sensor. 
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As with the dual-sensor towers, the AMU scripts checked the mean wind direction associated with 
each parameter from the single-sensor towers prior to every difference calculation for the towers with one 
sensor at a specified height . For onshore flow, the scripts verified the mean wind direction observed at 
the sensor was > 315° or < 135°, represented by the green arrows and arc in Figure 13. For offshore flow, 
the scripts verified the mean wind direction observed at the sensor was > 135° and <315°, represented by 
the red arrows and arc in Figure 13. 



Figure 13. As in Figure 11 but for the single-sensor towers. A 
single sensor was used for both onshore and offshore 
stratifications. 


4.3 Mean Monthly Model Bias Calculations 

The mean monthly model bias statistics were calculated using the previously computed hourly 
differences between the model forecast parameters and tower observation parameters. As discussed in 
Section 4.2, the model bias statistics were calculated in the master Excel workbook files and displayed in 
columns C-F as shown in Figure 1 5 by using the following equation: 


Bids f^esoN AM Monthly Mean 



Where: 


n = number of available model forecasts in any given stratification, 

/ = MesoNAM forecast of T, T d , wind speed or wind direction, and 
o = observed T, T d , wind speed or wind direction from each tower/sensor height. 


4.4 Standard Deviation of Mean Monthly Bias Calculations 

The model standard deviation of mean monthly bias statistics were calculated using the previously 
computed mean monthly bias and using the following equation: 


ST DEV MesoNAM Bias ~ 


£(* - X ) 2 


Where: 

n = number of available model forecasts in any given stratification, 
x = model bias of each forecast, 

x = mean bias of each forecast period in any given stratification ( Section 4.3). 
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4.5 Root Mean Square Error Calculations 

The model root mean square error (RMSE) was calculated using the following equations: 

RMSE = fiMSE 

n 

MSE = ^Jfi~°i¥ 

i = 1 

Where: 

n = number of available model forecasts in any given stratification, 

/ = MesoNAM forecast of T, T d , wind speed or wind direction, and 

o = observed T, T d , wind speed or wind direction from each tower/sensor height. 

4.6 Hypothesis Zero Test 

Hypothesis testing uses statistics to determine the probability that a given hypothesis is true. The 
AMU worked with the 45 WS to determine the best way to setup the test. The goal was to determine if the 
model bias of any of the parameters assessed throughout the model forecast period was not statistically 
significantly different from zero. The 45 WS recommended the test statistic for this data set would be to 
divide the mean bias of a given parameter by its mean standard deviation divided by the square root of the 
number of observations within a given stratification (onshore or offshore), as shown by: 

Mean 


Where 

Mean Bias as defined in Section 4.3, 

Mean STDEV as defined in Section 4.4, and 
n = number of observations. 

If this equation produced a value > -1.96 or < 1.96 (personal communication, Mr. Roeder, 45 WS) for a 
data point, then the bias at that point was not statistically significantly different from zero and the model 
forecast for that point was considered to have no error. 

The AMU worked closely with the 45 WS LWOs to determine the best way to display the results of 
the hypothesis zero test graphically so it would be useful for operations. The group agreed to have the 
hypothesis zero test displayed on the model bias charts as shaded regions only when the test was true 
(-1.96 < test value < 1.96). When the test was true, it indicated that even though the bias may not have 
been zero at any given data point, statistically the bias was zero. 

In Figure 14, there are several time periods when the hypothesis zero test was true as indicated by the 
green shading. The shape of the green shading is defined by the test value when the test was true. For an 
operational example, consider the forecast valid time 73-83 hr indicated by a red ellipse in Figure 14. 
Even though the model temperature bias (blue line) was not zero, the hypothesis zero test indicated that 
statistically, the temperature bias was zero and the LWO would not have to adjust the model temperature 
forecast for this time period. When the hypothesis zero test was false, the graph was not shaded and the 
LWO would have to consider adjusting the model forecast temperature. An example of a false hypothesis 
zero test is shown by a purple ellipse in Figure 14 during the 61-72 hr forecast valid time. 


Bias 




'Mean STDEV 


•fin 
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Model T Bias (Onshore) 

12Z Initialization, Tower 2, 54 ft, November 


■ Hypothesis Zero T (°F) 



Figure 14. An example of a verification chart showing model bias of T (blue line) and hypothesis zero 
test = true (light green shaded regions) from a 12 UTC model initialization using observations from 
Tower 2 at 54 ft for November with an onshore stratification. The red and purple ellipses highlight the 
region of the chart in the example discussed in the text. 

4.7 Statistics Values 

The AMU developed master files in Excel workbook format stratified by tower, sensor height, 
onshore/offshore flow and month in order to calculate the statistics. Figure 15 shows a portion of one of 
the master files for Tower 006 at 54 ft during offshore flow in June 2007-2009. Each master file contains 
four spreadsheets representing the model initialization times of 00, 06, 12 and 18 UTC. In each 
spreadsheet, column A shows the month, tower/sensor identification, sensor height and model 
initialization time and column B shows the model forecast hour, 0-84. Columns C-F contain the model 
bias, columns H-K contain the model standard deviation of the bias, and columns M-P contain the model 
RMSE for the month for each model forecast hour based on the formulas in each cell. The columns 
beginning with R contain the differences for each parameter that were previously calculated for each day 
of every month in the POR and each model forecast hour. 
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Figure 15. Example of a master Excel workbook file used to calculate the mean monthly statistics. 
Column A contains tower, sensor identification and model initialization information, column B shows the 
model forecast hour, columns C-P include the calculated statistics and the rest of the spreadsheet holds 
the hourly difference calculations. 


The AMU created charts in each spreadsheet that were automatically populated with the statistics by a 
VBA script that extracted the difference calculations from the merged spreadsheets and saved them in the 
master workbook files. Figure 16 shows the same spreadsheet as in Figure 15 but it is displaying several 
of the resulting charts based on the statistics calculated within the spreadsheet. The charts in the master 
workbooks were ultimately used to populate the graphical user interface (GUI) described in Section 6. 
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Figure 16 . Example of a master Excel workbook file used to calculate the mean monthly statistics. The 
graphs were automatically populated by the statistics calculated in the spreadsheet and show, from top to 
bottom, model bias, model standard deviation of bias and model root mean square error. 
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5. Model Performance 

The goal of this work was to provide the LWOs with a tool to evaluate model performance compared 
to observations at the tower(s) used to support a specific launch vehicle and not to evaluate the overall 
model performance at KSC/CCAFS. However, it is worth mentioning several characteristics of the model 
that were noted during this work. 

5.1 Accuracy 

As one would expect, the model’s accuracy decreased as the forecast valid time from the model 
initialization increased. Figure 17a and Figure 17b show examples of the model’s standard deviations of 
wind speed bias and wind direction bias, respectively. The trend line in both figures illustrates the 
increasing error in the model forecast compared to the wind observations. This model characteristic was 
noted in most of the data. 



Figure 17. An example of verification charts illustrating the model’s increasing error (solid black trend 
lines) for (a) the model standard deviation of wind speed bias (green line) and (b) standard deviation of 
wind direction bias (orange line) using observations from Tower 002 at 54 ft for October with an offshore 
stratification. 

5.2 Diurnal Signal 

A diurnal signal was present in the T and T d model bias as shown by the examples in Figure 18 and 
Figure 19. The diurnal signal was pervasive among the towers at varying degrees of amplitude. In 
general, the model had a cool T bias during the late night (~ 05-07 UTC) and a warm T bias during the 
afternoon (~ 17-19 UTC). The T d showed the opposite - the high bias during the late night and low bias 
during the afternoon. 

Figure 18a illustrates an example of the T bias at Tower 006 for a model initialization time of 00 
UTC, while Figure 1 8b is an example of the T bias for a model initialization time of 1 2 UTC for the same 
tower, sensor height, month and onshore stratification. These charts show that the model’s warm bias 
peaked at the 1 8, 42 and 66 hr forecast valid times from the 00 UTC initialization, and at the 6, 30, 54 and 
78 hr forecast valid times from the 12 UTC initialization. For both initialization times, the warm bias 
maxima occurred at about 1 8 UTC throughout the forecast. Conversely, the cool bias minima were at the 
29, 53 and 77 hr forecast valid times from the 00 UTC initialization, and at the 17, 41 and 65 hr forecast 
valid times from the 12 UTC initialization. For both initialization times, the cool bias minima occurred at 
about 06 UTC throughout the forecast. 
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Model T Bias (Onshore) 

00Z Initialization, Tower 6, 6 ft. May 


■■■Hypothesis Zero — — T(*F) 



Model T Bias (Onshore) 

12Z Initialization, Tower 6, 6 ft, May 


mb Hypothesis Zero T(*F) 



Figure 18. An example of verification charts illustrating the model’s diurnal signal for the model bias of 
T (blue line) and hypothesis zero test = true (light green shaded regions) (a) from a 00 UTC model 
initialization and (b) from a 12 UTC model initialization using observations from Tower 006 at 6 ft for 
May with an onshore stratification. 


Examples of the model’s diurnal T d signal is shown in Figure 19a and Figure 19b and are valid for the 
same tower, sensor height, month and onshore stratification as in Figure 18. While showing a similar 
pattern to T, the T d high and low biases occurred during the late night and afternoon, respectively, which 
was opposite of T. Figure 19a illustrates an example of the T d bias for a model initialization time of 00 
UTC, while Figure 19b is an example of the T d bias for a model initialization time of 12 UTC. 


Model Td Bias (Onshore) 

00Z Initialization, Tower 6, 6 ft, May 

■■Hypothesis Zero — Td(*F) 


Model Td Bias (Onshore) 

12Z Initialization, Tower 6, 6 ft. May 

■■Hypothesis Zero — Td(*F) 




Figure 19. As in Figure 1 8 but for T d . 


5.3 Model Parameter Vertical Level and Sensor Height 

The MesoNAM outputs forecast temperature and winds at heights of 2 m (~ 7 ft) and 10 m (~ 33 ft), 
respectively. In general, the model bias of T was smaller at a tower sensor height of 6 ft and larger at 
higher levels. The charts in Figure 20 illustrate this showing the model bias of T < |2.4 °F| at 6 ft (Figure 
20a) and the bias of T < |4.0 °F| at 54 ft (Figure 20b) at Tower 002. 


22 




Figure 20. An example of verification charts illustrating how the model’s bias varies with respect to 
tower sensor height for the model bias of T (blue line) and hypothesis zero test = true (light green shaded 
regions) from (a) the 6 ft sensor height and (b) the 54 ft sensor height using observations from Tower 002 
for June with an onshore stratification. 

Figure 21 illustrates how the model bias of wind speed differs at two different levels on Tower 002. 
At the higher sensor height of 54 ft (Figure 21a) the wind speed bias was < 5 kt while at the lower sensor 
height of 12 ft (Figure 21b) the bias was < 8 kt. In general, the wind speed bias was smaller at tower 
sensor heights above 54 ft and larger at 12ft. 



Figure 21. An example of verification charts illustrating how the model’s bias varies with respect to 
tower sensor height for (a) the model bias of wind speed (green line) from the 54 ft sensor height and (b) 
the 12 ft sensor height using observations from Tower 2 for June with an onshore stratification. 
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6. MesoNAM Verification Tool Graphical User Interface 

The dataset for this work contains 2,496 Excel spreadsheets with 4,896 charts. In order to present the 
data to the LWOs in a manageable and user-friendly manner, the AMU developed the MesoNAM 
Verification Tool using a Hyper-Text Markup Language (HTML) GUI so the tool could be used in most 
popular web browsers with computers running different operating systems such as Microsoft Windows 
and Linux. 

For easy navigation, the AMU implemented a multi-level drop-down menu written in JavaScript 
embedded within the HTML code. The main page of the GUI is shown in Figure 22. The title at the top of 
the page defines the name, version and date of the tool. The drop-down navigation menu is located under 
the title and allows the user to choose data by placing their mouse pointer over the desired tower/site to 
display a drop-down menu. A map of KSC/CCAFS showing the locations of the wind towers used to 
verify MesoNAM forecasts, the locations of the MesoNAM model grid points, the CCAFS weather 
station (KXMR) and the MesoNAM land/sea mask is shown as a reference for the LWOs. 



Figure 22. Main page of the GUI in a web browser showing the basic layout of the GUI. 
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An illustration of the drop-down menu is shown in Figure 23. Assume a user placed their mouse 
pointer over the menu item for Tower 002. By doing so, this displayed the first level of the drop-down 
menu, the months January- December. The user then moved the mouse pointer over November, the 12Z 
model initialization time, 54 ft sensor height, and then clicked "Onshore" for the flow. 



Figure 23. Main page of the GUI illustrating the functionality of the drop-down menu (heavy blue 
polygon region). 

The resulting page in the GUI displayed from these choices is shown in Figure 24. It contains charts 
showing the model bias, standard deviation of bias and RMSE of T, T d , wind speed and wind direction for 
onshore flow at a sensor height of 54 ft based on a 12 UTC model initialization for November at Tower 
002. The LWO can view all of the charts by using the vertical scroll bar on the right side of the web page. 
On each web page in the tool, the AMU added a link to the corresponding Excel workbook file that 
produced the charts on that page. This feature allows the user to view the spreadsheets from which the 
charts were derived. As shown in Figure 24, there is an Excel icon (red ellipse) just under the "Home" 
link on the navigation menu. When the user clicks the link. Excel opens the workbook file associated with 
the charts on that web page. 
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The AMU included a "Help/About" page as part of the GUI that briefly explains why it was 
developed, how to use it, sensor configuration and a description of the data (Figure 25). The link to this 
page is on the far right of the drop-down menu bar on the main page. The Help page includes sections 
describing the final report and how to access it, background information on the task, how to use the tool 
and the data used. 



Figure 24. A page from the GUI displaying four charts of model T bias and standard deviation of bias 
and model T d bias and standard deviation of bias for Tower 002 at a sensor height of 54 ft for a 12 UTC 
model initialization for November. The blue lines characterize T and the red lines T d . The green shaded 
regions correspond to a “true” hypothesis zero test. The icon representing a link to access the Excel file 
used to create the charts is shown within the red ellipse under the blue menu bar. 
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Figure 25. The “Help/About” page from the GUI. 
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7. Summary and Conclusions 

The AMU conducted an objective analysis of the MesoNAM forecasts compared to observed values 
from sensors at specified KSC/CCAFS wind towers by calculating the following statistics to verily the 
performance of the model: 

• Bias (mean difference), 

• Standard deviation of Bias, 

• Root Mean Square Error (RMSE), and 

• Hypothesis test for Bias = 0. 

The 45 WS LWOs use the MesoNAM to support launch weather operations. However, the actual 
performance of the model at KSC and CCAFS had not been measured objectively. The analysis compared 
the MesoNAM forecast winds, temperature and dew point to the observed values from the sensors on 
wind towers. The data were stratified by tower sensor, month and onshore/offshore wind direction based 
on the orientation of the coastline to each tower’s location. The model’s performance statistics were then 
calculated for each wind tower based on sensor height and model initialization time. The period of record 
for the data used in this task was based on the operational start of the current MesoNAM in mid-August 
2006 and so the task began with the first full month of data, September 2006, through May 2010. 

The analysis of model performance indicated: 

• The accuracy decreased as the forecast valid time from the model initialization increased, 

• There was a diurnal signal in T with a cool bias during the late night and a warm bias during the 
afternoon, 

• There was a diurnal signal in T d with a low bias during the afternoon and a high bias during the 
late night, and 

• The model parameters at each vertical level most closely matched the observed parameters at 
heights closest to those vertical levels. 

The AMU developed a GUI that consists of a multi-level drop-down menu written in JavaScript 
embedded within the HTML code. This tool allows the LWO to easily and efficiently navigate among the 
charts and spreadsheet files containing the model performance statistics. The objective statistics give the 
LWOs knowledge of the model’s strengths and weaknesses and the GUI allows quick access to the data 
which will result in improved forecasts for operations. 
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List of Acronyms 


45 WS 

45th Weather Squadron 

MesoNAM 12-km resolution NAM 

AMU 

Applied Meteorology Unit 

NAM 

North American Mesoscale 

CCAFS 

Cape Canaveral Air Force Station 

MSE 

Mean Square Error 

GUI 

Graphical User Interface 

NM 

Nautical Mile 

HTML 

Hyper-Text Markup Language 

POR 

Period of Record 

KTTS 

Shuttle Landing Facility 4-letter 
identifier 

QC 

RMSE 

Quality Control 
Root Mean Square Error 

KSC 

KXMR 

LWO 

Kennedy Space Center 

CCAFS rawinsonde 4-letter identifier 

Launch Weather Officer 

WRF 

Weather Research and Forecasting 
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NOTICE 


Mention of a copyrighted, trademarked or proprietary product, service, or document does not constitute 
endorsement thereof by the author, ENSCO Inc., the AMU, the National Aeronautics and Space 
Administration, or the United States Government. Any such mention is solely for the purpose of fully 
informing the reader of the resources used to conduct the work reported herein. 
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